System and method for obtaining thermal image data of a body part and thermal imager

ABSTRACT

A computer-implemented method for obtaining thermal image data of a body part and a thermal imager are disclosed. The method comprises receiving image data of a body part, the image data including thermal image data, the thermal image data including thermal image data of the body part and background thermal image data, applying a classifier to the image data to detect a profile of the body part, the classifier being configured to use image data other than the thermal image data to perform the detection, generating a mask from said profile, and applying the mask to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a 35 U.S.C. § 371 National Stage Patent Applicationof, and claims priority to, Patent Cooperation Treaty Application numberPCT/GB2019/051984, filed on 16 Jul. 2019, and entitled “SYSTEM ANDMETHOD FOR OBTAINING THERMAL IMAGE DATA OF A BODY PART AND THERMALIMAGER,” which claims priority to and the benefit of Great BritainPatent Application Number 1811620.2 filed on 16 Jul. 2018, where both ofthese applications are incorporated herein by reference in theirentirety.

FIELD OF THE INVENTION

The present invention relates to a system and method for obtainingthermal image data of a body part and to a thermal imager that isparticularly applicable for use in identifying diabetic foot ulceration.

BACKGROUND TO THE INVENTION

People with diabetes are prone to serious ulcers in their feet, whichcan become infected and ultimately require amputation. Early detectionis critical to improve patient outcomes, enabling doctors to interveneto protect the foot before skin breakdown occurs and, in cases ofinfection, to give antibiotics promptly.

It is estimated that up to 50% of diabetics experience foot neuropathywhich can result in undetected extensive skin damage, leading toulceration (diabetic foot ulceration (DFU)), infection and amputation.This train of events is largely avoidable with an estimated 80%reduction in amputations if early preventative action is taken.Mortality rates after DFU and amputation are high, with up to 70% ofpeople dying within 5 years of having an amputation and around 50% dyingwithin 5 years of developing a diabetic foot ulcer. Besides the adverseeffect on patients, it has been estimated that DFU and contingentcomplications cost the National Health Service in England and Wales inexcess of £1 billion per year.

Medical studies have identified that there is a persistent increase inskin temperature of >2° C. for up to 10 days before skin breakdown takesplace. Typically, before there are any visible signs of infection orabnormality in the feet, the skin temperature rises. This is notnecessarily infection-related and can also be caused by tissue breakingdown from friction and the subsequent increased blood flow to the area.

Achieving measurements of the necessary accuracy and being able tomonitor the temperatures of the foot over time is a particularchallenge.

Most physicians routinely examine a diabetic patient's feet visually,test for touch sensitivity, and palpate them to detect local temperaturevariations possibly indicating an incipient lesion (pre-ulceration).These manual techniques are notoriously inaccurate and can, where thereis concern over ulceration, be supplemented by thermal mapping (in whichthe foot contacts a measurement device or platform for temperaturemeasurement) or thermal imaging diagnostic techniques. However,commercial off the shelf thermal mapping and thermal imaging devices andtechniques also have various shortcomings.

Firstly, there are no known devices that have size, cost and ease of usesuitable for use outside of a large medical practice or laboratory.Ideally, such devices should be available for use by podiatrists, carehomes and the like without needing highly skilled operators.

Secondly, even with highly skilled operators, existing devices are notable to provide temperatures with clinically relevant uncertainties(i.e. better than ±0.3° C. (k=2). For example, a number of DFU currentthermal mapping devices require actual contact of the patient's feetwith the device. In such a case (contact-based measurement) the device'stemperature and thermophysical properties will perturb the object's(foot) temperature and will therefore impact accuracy of measurements.

Even when other areas of potential variance and error have beenminimised or eliminated, it has been determined that accuracy ofcommercial off the shelf thermal imagers used for absolute temperaturemeasurement varies considerably. In one study (“Performance tests ofthermal imaging systems to assess their suitability for quantitativetemperature Measurements” by Whittam et al, 12th InternationalConference on Quantitative Infrared Thermography, France, Bordeaux, 7-11Jul. 2014, https://www.ndt.net/search/docs.php3?showForm=off&id=17768),it was identified that five out of six thermal imaging devices testeddemonstrated errors in temperature measurement that fell outside of themanufacturer's own stated error margin at a distance of 1 m from themeasurement target. Given that the manufacturer's stated error marginwas typically ±2° C. and the thermal imaging devices could notrepeatedly and uniformly achieve this error margin, they are not capableof providing clinically relevant temperatures (uncertainties<±0.3° C.)and it is highly questionable whether they could be relied upon to givean early warning of a persistent increase in skin temperature of >2° C.

Reliable foot temperature comparisons over time require accurate spatialregistration of the foot each time an image is obtained. This can bechallenging because operators (and their abilities and approachesemployed) may differ and foot position and size may change betweenpatients and between measurements, distorting the foot temperaturemeasurement accuracy.

There therefore appears substantial room for improvement of devices andapproaches. If the temperature increase associated with neuropathy couldbe accurately identified in a timely manner it is expected that itshould be straightforward to greatly reduce ulceration through (forexample) preventative off-loading.

STATEMENT OF INVENTION

According to an aspect of the present invention, there is provided acomputer-implemented method for obtaining thermal image data of a bodypart comprising:

receiving image data of a body part, the image data including thermalimage data, the thermal image data including thermal image data of thebody part and background thermal image data;

applying a classifier to the image data to detect a profile of the bodypart, the classifier being configured to use image data other than thethermal image data to perform the detection;

generating a mask from said profile;

applying the mask to the thermal image data to remove the backgroundthermal image data and obtain thermal image data of the body part.

Embodiments of the present invention concern a method and system inwhich additional data is used to guide extraction of thermal images ofbody parts. The additional data may be other image data (such as visualimage or a stereo visual image) of the same scene. The additional datamay also (or alternatively) include training data that has been used totrain a machine learning or artificial intelligence system such as aneural network, genetic algorithm or the like. For example, a trainingbank of images of feet (visual, thermal images, other image types orsome combination) can be used to train the classifier to determine aprofile from presented image data and that profile can then be used togenerate a mask and extract the thermal image data of the body part. Inpreferred embodiments, the system is used in a hand-held thermal imagingsystem for identifying potential or predicted foot ulceration fromcaptured thermal imagery.

In thermal imagers, there is a general assumption that the foregroundimaged object will be much hotter than the surrounding environment(background) so that background subtraction can be applied. However,this assumption does not always hold true. Background temperaturescannot be guaranteed to differ substantially from the foreground bodypart and may vary depending on time of day, season or environment (forexample it is not uncommon in hospitals and care homes where embodimentsof the invention may be used to have relatively high backgroundtemperatures to provide comfort to those who have limited movement orare bed ridden). Additionally, DFU's are common around the big toe andball of the foot. Where such areas are cooler (due to circulationissues, for example), there is a risk that areas of the foot may not beproperly differentiated from the background. Embodiments of the presentinvention seek to address this by utilising additional image data tomore clearly differentiate edges of the body parts.

It will be appreciated that approaches may be operated in parallel or inconjunction, for example a neural network producing what it considers tobe the profile from the thermal image data and this being contrastedwith the profile generated from a machine vision system from stereoimage of the scene. Alternatively, the neural network could operate onthe stereo image and the resultant profile then applied to the thermalimage.

The image data may include visual image data captured from the samescene as the thermal image data, the classifier being configured todetect the profile of the body part from the visual image data.

The image data may include visual stereo image data captured from thesame scene as the thermal image data.

The classifier may comprise an artificial neural network or otherartificial intelligence system trained from image data of body parts todetect the profile of the body part.

The classifier may be configured to determine a spatial orientation ofthe body part, the method further including the step of modifyingspatial orientation of the obtained thermal image data of the body partin dependence on the determined spatial orientation to match a defaultspatial orientation.

Although much of the current focus is on thermal imaging for detectingonset of foot neuropathy and triggering advanced preventative measures,the approaches set out in this application are also applicable to otherbody parts such as eyes, limbs and other body parts of human and/oranimal subjects.

The inventors of the present application have been conceived anon-contact thermal imaging device, system and method that addressesmany of the shortcomings of known thermal imaging diagnostic devices andsystems.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described by way ofexample only with reference to the accompanying drawings in which:

FIG. 1 is a schematic diagram of a thermal imager according to anembodiment of the present invention;

FIG. 2 is a schematic diagram of a thermal image processing systemaccording to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating aspects of the system of FIG.2 in more detail;

FIG. 4 is a schematic diagram illustrating aspects of the system of FIG.2 in more detail; and,

FIG. 5 is a schematic diagram illustrating alternate aspects of thesystem of FIG. 2 in more detail.

DETAILED DESCRIPTION

FIG. 1 is a schematic diagram of a thermal imager according to anembodiment of the present invention.

The thermal imager 10 is preferably of a hand-held form factor andincludes a thermal camera 20, a battery power source 30, a processor 40,a data repository 50 and a touchscreen display 60.

The thermal imager also includes a thermal image processing system 100that is executed by the processor 40 to obtain thermal image data on abody part from raw thermal image data captured by the thermal camera 20.Further details on the thermal image processing system are set out belowwith reference to FIGS. 2-4 .

In operation, the user positions the thermal imager such that the bodypart to be imaged is within the field of view of the thermal camera.Output of the thermal camera may be displayed on-screen to aid this.

Preferably, the thermal imager provides the user with a live scenedisplay and the imager is orientated by the user such that the object ofinterest (foot) is in image. Alignment tools are preferably used such asa distance pointer/on screen overlays (akin to visual camera systems)etc. When the user is happy with orientation a button is depressed (orscreen tap) and an image is captured.

Optionally, alignment guides may be projected from the device usingLED's or the like. In such an arrangement, the user points the alignmentguides at specific features of the target area(s) of the body part oralternatively positions the body part/thermal imager such that the bodypart falls inside the projected alignment guides.

A thermal image of the body part is then produced by the thermal imageprocessing system 100 and written to the data repository 50. This may bedisplayed on screen, compared to prior thermal images of the body partin the data repository 50, to an image of the opposite foot and/or usedto trigger an alert and/or highlight areas of potential concernexhibiting temperatures indicative of inflammation.

Where inflammation may arise in different parts/sides of a body part,the thermal imager 10 may include capture protocols in the form ofcomputer program instructions that are executed by the processor 40 andguide the user (for example by on-screen prompts on the display 60) tofollow a predetermined measurement protocol to capture predeterminedviews of the body part. In the case of a foot, these view may include(in no particular order): left foot (sole—plantar), right foot(sole—plantar), left foot (top—dorsal), right foot (top—dorsal), leftfoot (outer side—lateral), right foot (outer side—lateral), left foot(inner side—medial), right foot (inner side—medial).

Preferably, the thermal imager is a micro-bolometer array operating at8-14 μm. The micro-bolometer array is preferably a staring focal planearray in which its pixel elements are micro-bolometers that aresensitive to infrared radiation/temperature. In one embodiment, thepixels are each 17-25 μm in size. Infrared radiation is emitted by everyobject above absolute 0. Infrared radiation has a strong (T⁴)relationship with temperature—such that temperature of an object can bedetermined from measuring its emitted infrared radiation. A thermalimage is captured by measuring the infrared radiation received at eachpixel of the array. This data may be encoded/compressed for storageefficiency (such as run length encoding) and can be handled using imageprocessing in a similar way to a pixel-based visual image.

It will be appreciated that the touchscreen display could be replaced byanother user interface such as a display and keypad or it could bereplaced by some other arrangement such as a Bluetooth connection to asmartphone or the like (which then acts as a remote user interface forcontrolling the thermal imaging system 10 and viewing imagery).

The thermal imager 10 may also include an I/O system for wired orwireless communication of obtained thermal images or other data. Thebattery power source 30 may be rechargeable and/or replaceable.

FIG. 2 is a schematic diagram of a thermal image processing system forobtaining thermal image data of a body part according to an embodimentof the present invention.

The thermal image processing system 100 is configured to receive thermalimage data 110. The thermal image data includes both thermal image dataof the body part 111 and background thermal image data 112.

In order to extract the thermal image data of the body part 111, thethermal image processing system 100 applies a classifier 120 to detect aprofile of the body part. The classifier uses image data 130 other thanthe thermal image data 110 to perform the detection, examples of whichare described with reference to FIGS. 3 and 4 below.

Once a profile has been detected, it is used by the thermal imageprocessing system to generate a mask 113. The thermal image processingsystem then applies the mask 113 to the thermal image data 110 to removethe background thermal image data 112 and obtain thermal image data ofthe body part 111.

FIG. 3 is a schematic diagram illustrating aspects of the system of FIG.2 in more detail.

In one embodiment, the thermal imager may also include a visual imagingsensor 70 that is configured to capture the visual spectrum of theimaged scene to generate visual image data 130 a (this may be storedseparately to the thermal image data 110 or they may be storedcollectively). The visual imaging sensor 70 may capture an RGB image ofthe scene, for example.

FIG. 4 is a schematic diagram illustrating aspects of the system of FIG.2 in more detail.

In a preferred embodiment, the thermal imager includes a stereo visualimaging sensor 80 (in one embodiment this is a pair of visual imagingsensors having a predetermined orientation and distance therebetweensuch that stereo image processing can be performed).

The stereo visual imaging sensor 80 is configured to capture a stereoimage of the visual spectrum of the imaged scene to generate visualimage data 130 b (as before, this may be stored separately to thethermal image data 110 or they may be stored collectively). The visualimaging sensor 80 may capture a pair of RGB images of the scene, forexample. In this embodiment, the classifier 120 uses the stereo visualimage data 130 b to detect the profile of the body part from the visualimage data.

The classifier may optionally be configured to determine a spatialorientation of the body part from the image data 130. For example, theclassifier may use the stereo visual image data to determine spatialorientation of the body part and the thermal image processing system maythen modify spatial orientation of the obtained thermal image data ofthe body part in dependence on the determined spatial orientation tomatch a default spatial orientation.

In one embodiment, the thermal image processing system 100 may use thestereo visual image data 130 b to generate a 3D model (preferably a 3Dtopography mesh) of the body part onto which the thermal imaging data isregistered. The 3D model may be generated using photometric stereo,structure from motion or other approaches.

Further data may be overlaid onto the 3D model such as interfacialpressure data, thermal measurements from the thermal image data such asMax, Min, Mean, Span, Histogram etc.

FIG. 5 is a schematic diagram illustrating alternate aspects of thesystem of FIG. 2 in more detail.

In this embodiment, the classifier 120 includes an artificial neuralnetwork that has been trained from image data 130 other than the thermalimage data including image data of body parts and is configured todetect the profile of the body part. In this embodiment, the image dataother than the thermal image data may or may not have been of thatparticular person's body part and may include thermal image data, visualimage data or other image data (or a combination).

The neural network may, for example, be a convolutional neural network.The artificial neural network may be self-organising, feed-forward

It is to be appreciated that certain embodiments of the invention asdiscussed above may be incorporated as code (e.g., a software algorithmor program) residing in firmware and/or on computer useable mediumhaving control logic for enabling execution on a computer system havinga computer processor. Such a computer system typically includes memorystorage configured to provide output from execution of the code whichconfigures a processor in accordance with the execution. The code can bearranged as firmware or software, and can be organized as a set ofmodules such as discrete code modules, function calls, procedure callsor objects in an object-oriented programming environment. If implementedusing modules, the code can comprise a single module or a plurality ofmodules that operate in cooperation with one another.

Optional embodiments of the invention can be understood as including theparts, elements and features referred to or indicated herein,individually or collectively, in any or all combinations of two or moreof the parts, elements or features, and wherein specific integers arementioned herein which have known equivalents in the art to which theinvention relates, such known equivalents are deemed to be incorporatedherein as if individually set forth.

Although illustrated embodiments of the present invention have beendescribed, it should be understood that various changes, substitutions,and alterations can be made by one of ordinary skill in the art withoutdeparting from the present invention which is defined by the recitationsin the claims below and equivalents thereof.

The invention claimed is:
 1. A computer-implemented method for obtainingthermal image data of a body part, the method being executed in ahandheld device including a display and a thermal imager and comprising:receiving image data of a body part, the image data including thermalimage data, the thermal image data including thermal image data of thebody part and background thermal image data; applying a classifier tothe image data to detect a profile of the body part, the classifierbeing configured to use image data other than the thermal image data toperform the detection; generating a mask from said profile; applying themask to the thermal image data to remove the background thermal imagedata and obtain thermal image data of the body part, wherein the imagedata includes visual image data captured from the same scene as thethermal image data, the classifier detecting the profile of the bodypart from the visual image data, wherein the image data includes visualstereo image data captured from the same scene as the thermal imagedata, the method further comprising displaying via the display a livescene display of the scene to be captured by the thermal camera to guidepositioning of the handheld device by the user such that the body partis in image.
 2. The method of claim 1, wherein the step of applying aclassifier includes applying an artificial neural network trained fromimage data of body parts to the received image data to detect theprofile of the body part.
 3. The method of claim 1, further comprisingdetermining a spatial orientation of the body part by the classifierfrom the visual stereo image data and modifying spatial orientation ofthe obtained thermal image data of the body part in dependence on thedetermined spatial orientation to match a default spatial orientation.4. A handheld thermal imager including a thermal camera, a display, anda processor, the processor being configured to execute computer programcode for executing a classifier, including: computer program codeconfigured to receive image data of a body part, the image dataincluding thermal image data from the thermal camera, the thermal imagedata including thermal image data of the body part and backgroundthermal image data; computer program code configured to apply aclassifier to the image data to detect a profile of the body part, theclassifier being configured to use image data other than the thermalimage data to perform the detection; computer program code configured togenerate a mask from said profile; computer program code to apply themask to the thermal image data to remove the background thermal imagedata and obtain thermal image data of the body part, further comprisinga visual imager configured to capture visual image data on the samescene as the thermal image data, the computer program code configured toapply the classifier detecting the profile of the body part from thevisual image data, wherein the visual imager includes a stereo visualimager, the display configured to display a live scene display of thescene to be captured by the thermal camera.
 5. The thermal imager ofclaim 4, wherein the computer program code configured to apply aclassifier includes computer program code configured to apply anartificial neural network trained from image data of body parts to thereceived image data to detect the profile of the body part.
 6. Thethermal imager of claim 4, wherein the processor is further configuredto execute computer program code to determine a spatial orientation ofthe body part by the classifier from stereo image data from the stereovisual imager and modify spatial orientation of the obtained thermalimage data of the body part in dependence on the determined spatialorientation to match a default spatial orientation.
 7. Acomputer-implemented method for obtaining thermal image data of a bodypart comprising: receiving image data of a body part, the image dataincluding thermal image data, the thermal image data including thermalimage data of the body part and background thermal image data; applyinga classifier to the image data to detect a profile of the body part, theclassifier being configured to use image data other than the thermalimage data to perform the detection; generating a mask from saidprofile; applying the mask to the thermal image data to remove thebackground thermal image data and obtain thermal image data of the bodypart, wherein the image data includes visual image data captured fromthe same scene as the thermal image data, the classifier detecting theprofile of the body part from the visual image data, wherein the imagedata includes visual stereo image data captured from the same scene asthe thermal image data, the method further comprising generating a 3Dmodel of the body part and projecting the thermal imaging data into the3D model.
 8. The thermal imager of claim 4, further comprising a visualalignment tool to guide alignment of the body part.
 9. The thermalimager of claim 8, wherein the visual alignment tool comprises anon-screen overlay on the display to guide alignment of the body part.10. The thermal imager of claim 8, wherein the visual alignment toolcomprises a light emitter configured to project visual alignment guidesfrom the handheld device to be positioned at or around the body part.11. A handheld thermal imager including a thermal camera, a visualimager configured to capture visual image data on the same scene asimaged by the thermal camera wherein the visual imager includes a stereovisual imager, and a processor, the processor being configured toexecute computer program code for executing a classifier, including:computer program code configured to receive image data of a body part,the image data including thermal image data from the thermal camera, thethermal image data including thermal image data of the body part andbackground thermal image data; computer program code configured to applya classifier to the image data to detect a profile of the body part, theclassifier being configured to use image data other than the thermalimage data to perform the detection; computer program code configured togenerate a mask from said profile; computer program code to apply themask to the thermal image data to remove the background thermal imagedata and obtain thermal image data of the body part, the computerprogram code configured to apply the classifier detecting the profile ofthe body part from the visual image data, the processor being furtherconfigured to execute computer program code to generate a 3D model ofthe body part and project the thermal imaging data into the 3D model.12. The method of claim 7, wherein the step of applying a classifierincludes applying an artificial neural network trained from image dataof body parts to the received image data to detect the profile of thebody part.
 13. The method of claim 7, further comprising determining aspatial orientation of the body part by the classifier from the visualstereo image data and modifying spatial orientation of the obtainedthermal image data of the body part in dependence on the determinedspatial orientation to match a default spatial orientation.